The subject matter discussed in the background section should not be assumed to be prior art merely as a result of its mention in the background section. Similarly, a problem mentioned in the background section or associated with the subject matter of the background section should not be assumed to have been previously recognized in the prior art. The subject matter in the background section merely represents different approaches, which in and of themselves may also correspond to the claimed embodiments.
There is a widely understood need to test code before releasing the code into production.
One approach for testing code, particularly in an on-demand system, is to create a development environment which is separate from the production environment and run tests in that development environment. In such a way, side-effects resulting from running the tests purportedly do not negatively influence the production environment. However, several problems exist with such a model. First, in an on-demand system, the development and production environments may use the same infrastructure. Hence, the development environment uses resources that could be used for production work, resulting in a less efficient production environment. Further, data in the development environment may not be identical to that within a live production environment. For example, there may always be delays between the data in the production environment and the copy represented within the development environment. Tests written to rely upon such production data may therefore fail in a development environment due to a lack of data fidelity. Tests written in a particular context, such as with the assumption of a certain level of data access, may also trigger a failure where access levels or context is not the same between a development environment and a production environment. Additional problems may arise because the physical infrastructure of the development environment is not precisely identical to the production environment. For example, the development environment may be scaled down, may be implemented on older or different hardware, may be subject to different resource demands which affect stress, heat, latency, etc.
A different approach is to perform tests within the live production environment using live production data. This approach is advocated by some working with database systems that implement transaction journaling and roll-back capability. The argument in support of such an approach is that any transaction against the database may be rolled back, so long as it is not committed, thus preventing negative side-effects to the database or the production environment. Notwithstanding such justification, this model is also flawed. For example, if a test sequence acts upon a data element of production data, such as a database field or row, that test sequence may cause a lock on the corresponding data element, which in turn may cause a data access failure for a production transaction. The resulting data access failure is undesirable as it preempts higher value production work from being completed. Further problems and undesirable side-effects from testing upon a production environment may occur where the test transacts against a large number of database elements, thus requiring a rollback to a large number of transactions. Such a rollback creates operational burden for the database which may negatively affect performance of production transactions which are being processed concurrently.
The present state of the art may therefore benefit from a testing data silo, including methods, systems, and apparatuses for creating a data silo and testing with a data silo in an on-demand service environment as described herein.